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DETAILED ACTION 

Response to Arguments 

1 . Applicant's arguments with respect to claims 1 -6, 9, 1 0, 1 2-1 4, 27, 28, and 
30-40 have been considered but are moot in view of the new ground(s) of 
rejection. 

Claim Rejections - 35 USC § 103 

2. The text of those sections of Title 35, U.S. Code not included in this action 
can be found in a prior Office action. 

3. Claims 1-6, 9, 10, 12-24, 27, 28, and 30-40 are rejected under 35 
U.S.C. 103(a) as being unpatentable over Drews et al. (previously cited and 
hereinafter Drews), US 5,768,607 A, in view of Baumgartner et al. (previously cited 
and hereinafter Baumgartner), US 5,642,171 A, and Bullwinkel etal. (hereinafter 
Bullwinkel), US 6,099,317 A. 

4. Regarding claim 1, Drews teaches a method for synchronizing operations 
in a computer environment with accompanying audio (see column 3, line 15 - 
column 4, line 2), said method comprising: 

replaying sa/'d operations and said accompanying audio in said computer environment 
said operations resulting from processing of recorded user inputs... (see column 5, lines 
37-57, and column 7, line 12 - column 8, line 52); 

creating a synchronization point at a common point in said replaying of said operations 
and said accompanying audio (see column 13, lines 30-36); 

associating said synchronization point with said accompanying audio, said 
synchronization point providing a reference point to substantially synchronize said 
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accompanying audio when said operations are replayed in a replay computer environment 
using said recorded user inputs (see column 13, lines 30-36); 

detecting said synchronization point during a subsequent replay of said operations and 
said accompanying audio in said replay computer environment, said subsequent replay 
involving another processing of said recorded user inputs (see column 14, lines 4-26); 

comparing said synchronization point with a time value associated with said another 
processing of said recorded user inputs (see column 14, lines 4-48); 

Drews teaches a method, wherein operations with accompanying audio is 

synchronized in a computer environment. However, Drews does not teach 

pausing accompanying audio if the operations (i.e. recorded user inputs) fall 

behind the synchronization, so that they video can catch up to the audio 

portion of the recording. Therefore Drews does not teach these features: 

replaying said operations and said accompanying audio in said computer environment 
using an event recording, said operations resulting from processing of recorded user inputs, said 
event recording including ail user inputs to an original computer environment during said event 
recording and initial conditions of said original computer environment when said event 
recording was initiated; 



selectively pausing said subsequent replay of said accompanying audio if a difference 
between said synchronization point and said time value exceeds a predefined amount so that 
said subsequent replay of said operations can catch up to said accompanying audio; and 

resuming said subsequent replay of said accompanying audio if a difference between 
said synchronization point and a current time value does not exceed a second predefined 
amount, said current time value being associated with said another processing of said recorded 
user inputs 

Baumgartner teaches a method of synchronizing audio and video 
streams in a computer environment (see abstract). Specifically, Baumgartner 
teaches an environment, wherein the processing capabilities of the system are 
too slow (see column 5, lines 18-27). It is not hard for one of ordinary skill in the 
art at the time of the invention to appreciate that a video processor may have 
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the same deficiencies. Baumgartner teaches pausing the audio if the audio is 

too far ahead of the video playback (see column 7, lines 1 6-23). Specifically, 

Baumgartner teaches the selectively pausing and resuming steps (see figure 5A, 

steps 518, 520, 522, and 524 and column 14, lines 1-26). It would have been 

obvious for one of ordinary skill in the art at the time of the invention to combine 

the teachings of Drews and Baumgartner for the purpose of synchronizing audio 

and video (i.e. recorded user input) on slower computer hardware. However, 

the combination does not appear to teach: 

replaying said operations and said accompanying audio in said computer environment 
using an event recording, said operations resulting from processing of recorded user inputs, said 
event recording including all user inputs to an original computer environment during said event 
recording and initial conditions of said original computer environment when said event 
recording was initiated; 

Bullwinkel teaches a system for computer-based training (CBT) (see 

Bullwinkel, column 2, lines 1 1-26). Bullwinkel teaches recording all user inputs are 

during an event recording, or session (see Bullwinkel , column 1 , lines 33-38, 

column 6, line 55- column 7, line 1 1, and column 12, lines 5-18). Furthermore, 

Bullwinkel teaches that the "Application Player/Recorder" (APR) is initialized and 

a start state is recorded (see Bullwinkel, column 7, lines 5-1 1, column 12, lines 19- 

35, figure 5, step 802, and figure 10, step 302). One of ordinary skill in the art at 

the time of the invention would have found it obvious that the APR program uses 

a start state, so that the program to be taught to a user of the CBT system would 

be in the correct state. It would have been obvious for one of ordinary skill in 
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the art at the time of the invention to combine the teachings of Drews, 
Baumgartner, and Bullwinkel for the purpose of providing consistent results in a 
session recording playback. 

5. Regarding claim 2, the further limitation of claim 1 , see the preceding 
rejection with respect to claim 1 . The combination teaches the method of claim 
1, wherein 

creating of said synchronization point includes creating said synchronization point in 
response to a user command (see Drews, column 13, lines 30-36, wherein a user 
recording sound and cursor movement causes the creation of synchronization 
points). 

6. Regarding claim 3, the further limitation of claim 1 , see the preceding 
rejection with respect to claim 1 . The combination teaches the method of claim 
1, wherein 

said common point is at a point in time where there is no audio output during said 
replaying of said accompanying audio (see Drews, column 13, lines 37-45 and figure 
1 la, wherein common points are created at the beginning of recording, when 
there is no sound). 

7. Regarding claim 4, the further limitation of claim 1 , see the preceding 

rejection with respect to claim 1 . The combination teaches the method of claim 

1, further comprising 

obtaining a current time value associated with said processing of said recorded user 
inputs, said current time value corresponding to said synchronization point (see 
Baumgartner, figures 5a and 5b and column 13, lines 23-67, wherein the current 
time value is implied to be a current audio or video time index). 

8. Regarding claim 5, the further limitation of claim 1 , see the preceding 
rejection with respect to claim 1 . In the combination, Baumgartner teaches 
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different ways of storing the video and the audio (see column 10, lines 30-39). 
Baumgartner does not appear to teach a method of claim 1 , wherein the 
synchronization point is saved in the first file with the recorded audio, but it 
would have been obvious for one of ordinary skill in the art at the time of the 
invention to place the synchronization points in either file, or a single file with 
interleaved data. It would have been an obvious design choice. 

9. Regarding claim 6, the further limitation of claim 1 , see the preceding 
rejection with respect to claim 1 . The combination teaches the features of 
claim 1 , and makes these features obvious. In the combination, Drews teaches 
the editing of the user commands, or drawings, and editing of the recorded 
sound (see column 10, lines 30-40). It is inferred by the editing, or appending, of 
the recording will change a time value of the synchronization points. Drews 
teaches a timeline (see figure 11a, lib, and 12). It would have been obvious for 
one of ordinary skill in the art at the time of the invention to allow direct editing 
of the timeline by moving a positional marker on the timeline as is known in 
video editing. 

1 0. Regarding claim 9, the further limitation of claim 1 , see the preceding 
rejection with respect to claim 1 . The combination teaches the method of claim 
1, wherein 

said second predefined amount equals said predefined amount (see Baumgartner, 
column 14, lines 1-26 teaches one embodiment, wherein the predefined 
amounts are different). 
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In the combination, Baumgartner teaches a preferred embodiment, wherein it is 
obvious to change the predetermined amounts according to different 
applications. It would have been obvious for one of ordinary skill in the art at 
the time of the invention to use the same predefined amounts or thresholds to 
pause or resume the audio in order to make the comparison of the current time 
value with the synchronized playback value easier. 

1 1 . Regarding claim 10, see the preceding rejection with respect to claim 1 . 
The combination teaches the method of claim 1, which reads on the method of 
claim 10. 

12. Regarding claim 12, the further limitation of claim 10, see the preceding 
rejection with respect to claim 9. The combination makes obvious the method 
of claim 10 with these additional features. 

13. Regarding claim 13, the further limitation of claim 10, see the preceding 
rejection with respect to claim 6. The combination makes obvious the method 
of claim 10 with these additional features. 

14. Regarding claim 14, the further limitation of claim 10, see the preceding 
rejection with respect to claim 1 . The combination teaches the features of 
claim 10 and these additional features. 
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15. Regarding claim 15, the further limitation of claim 14, see the preceding 
rejection with respect to claim 2. The combination teaches the features of 
claim 14 and these additional features. 

1 6. Regarding claim 16, the further limitation of claim 14, see the preceding 
rejection with respect to claim 3. The combination teaches the features of 
claim 14 and these additional features. 

1 7. Regarding claim 17, the further limitation of claim 14, see the preceding 
rejection with respect to claim 5. The combination teaches the features of 
claim 14 and these additional features. 

18. Regarding claim 18, the further limitation of claim 14, see the preceding 
rejection with respect to claim 6. The combination makes obvious the method 
of claim 14 with these additional features. 

1 9. Regarding claim 1 9, see the preceding rejection with respect to claim 1 . 
The combination teaches the method of claim 1, which reads on the storage 
medium readable by a computer, tangibly embodying a program of 
instructions executable by said computer to perform method steps for 
synchronizing operations in a computer environment with accompanying audio, 
said method comprising the features of claim 1 . 

20. Regarding claim 20, the further limitation of claim 19, see the preceding 
rejection with respect to claim 2. The combination teaches the features of 
claim 2, which reads on the features of claim 20. 
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21 . Regarding claim 21, the further limitation of claim 19, see the preceding 
rejection with respect to claim 3. The combination teaches the features of 
claim 3, which reads on the features of claim 21 . 

22. Regarding claim 22, the further limitation of claim 19, see the preceding 
rejection with respect to claim 4. The combination teaches the features of 
claim 4, which reads on the features of claim 22. 

23. Regarding claim 23, the further limitation of claim 19, see the preceding 
rejection with respect to claim 5. The combination teaches the features of 
claim 4, which reads on the features of claim 23. 

24. Regarding claim 24, the further limitation of claim 19, see the preceding 
rejection with respect to claim 6. The combination makes obvious the method 
of claim 19 with these additional features. 

25. Regarding claim 27, the further limitation of claim 1 9, see the preceding 
rejection with respect to claim 9. The combination teaches the features of 
claim 4, which reads on the features of claim 27. 

26. Regarding claim 28, see the preceding rejection with respect to claim 1 . 
The combination teaches the method of claim 1, which reads on the storage 
medium readable by a computer, tangibly embodying a program of 
instructions executable by said computer to perform method steps for 
synchronizing operations in a computer environment with accompanying audio, 
said method comprising the features of claim 10. 
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27. Regarding claim 30, the further limitation of claim 28, see the preceding 
rejection with respect to claim 9. The combination makes obvious the storage 
medium of claim 28 with these additional features. 

28. Regarding claim 31, the further limitation of claim 28, see the preceding 
rejection with respect to claim 6. The combination makes obvious the method 
of claim 28 with these additional features. 

29. Regarding claim 32, the further limitation of claim 28, see the preceding 
rejection with respect to claim 1 . The combination teaches the features of 
claim 28 and these additional features. 

30. Regarding claim 33, the further limitation of claim 32, see the preceding 
rejection with respect to claim 2. The combination teaches the features of 
claim 32 and these additional features. 

31 . Regarding claim 34, the further limitation of claim 32, see the preceding 
rejection with respect to claim 3. The combination teaches the features of 
claim 32 and these additional features. 

32. Regarding claim 35, the further limitation of claim 32, see the preceding 
rejection with respect to claim 5. The combination teaches the features of 
claim 32 and these additional features. 

33. Regarding claim 36, the further limitation of claim 32, see the preceding 
rejection with respect to claim 6. The combination makes obvious the method 
of claim 32 with these additional features. 



Application/Control Number: 10/770,935 Page 1 1 

Art Unit: 2614 

34. Regarding claim 37, see the preceding rejection with respect to claim 1 6. 
The combination teaches a method with the combined features of claims 10, 
14, and 16. 

35. Regarding claim 38, see the preceding rejection with respect to claim 37. 
The combination teaches the method of claim 16, which reads on the storage 
medium readable by a computer, tangibly embodying a program of 
instructions executable by said computer to perform method steps for 
synchronizing operations in a computer environment with accompanying audio, 
said method comprising the features of claim 37. 

36. Regarding claim 39, see the preceding rejection with respect to claim 1 7. 
The combination teaches a method with the combined features of claims 10, 
14, and 17. 

37. Regarding claim 40, see the preceding rejection with respect to claim 39. 
The combination teaches the method of claim 1 7, which reads on the storage 
medium readable by a computer, tangibly embodying a program of 
instructions executable by said computer to perform method steps for 
synchronizing operations in a computer environment with accompanying audio, 
said method comprising the features of claim 39. 

38. Regarding claim 41, see the preceding rejection with respect to claim 1 . 

The combination makes obvious the method of claim 1, further comprising: 

saving said initial conditions of said original computer environment in a log file when a 
recording is initiated, said initial conditions corresponding to an initial state of said original 
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computer environment such that said initial state of said original computer environment can be 
automatically recreated on replay using said initial conditions, said initial state being a particular 
state from a plurality of possible states for said original computer environment, said log file 
including complete definitions of every control in said original computer environment with 
respect to said initial state so that said initial state can be subsequently recreated using said log 
file (see Bullwinkel, column 1 2, line 53 - column 1 4, line 30); 

modifying said initial conditions in said log file in response to user editing of said log file so 
that a modified initial state of said original computer environment is automatically created on 
replay using modified initial conditions in said log file when said log file is loaded (see 
Bullwinkel column 13, lines 32-65); and 

automatically loading said log file in said computer environment when a replay is 
initiated to create said modified initial state in said computer environment as a starting state for 
said replay (see Bullwinkel, column 5, lines 23-35 and column 10, lines 41-58). 

In the combination Bullwinkel teaches initial conditions corresponding to 

an initial state, wherein these teachings make obvious a log file, or the 

equivalent, for saving these conditions and states. Bullwinkel teaches a user 

editing, or appending, a FSM by adding actions corresponding to new states 

(see Bullwinkel column 1 3, lines 32-65). Last, it is obvious to one of ordinary skill in 

the art at the time to automatically load the log file when the play sequence is 

started for the purpose of an experienced user teaching a new user how to use 

a new program (see Bullwinkel column 5, lines 23-35 and column 10, lines 41-58). 

39. Regarding claim 42, see the preceding rejection with respect to claims 1 0 
and 41 . The combination makes obvious the method of claim 10, further 
comprising these features (see Bullwinkel, column 5, lines 23-35, column 10, lines 
41-58, and column 12, line 53 - column 14, line 30). 

40. Regarding claim 43, see the preceding rejection with respect to claims 1 9 
and 41 . The combination makes obvious the method of claim 19, further 
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comprising these features (see Bullwinkel, column 5, lines 23-35, column 10, lines 
41-58, and column 12, line 53 - column 14, line 30). 



Conclusion 

41 . The prior art made of record and not relied upon is considered pertinent 
to applicant's disclosure. 

Owen, US 5,585,583 A, (previously cited) teaches mixing a student's 
musical instrument with a synchronized audio/video playback of an instructional 
musical piece (abstract, column 3, line 57 - column 4, line 9, column 6, lines 43- 
59, column 7, lines 15-26, column 7, line 56 - column 8, line 16 and column 13, 
lines 48); 

Kuribayashi, US 5,615,189 A, (previously cited) teaches synchronization of a 
word clock in MIDI circuits (abstract and column 1, line 50 - column 2, line 20); 

Berstis et al., US 5,903,266 A, (previously cited) teaches audio instructions 
detailing various steps of connecting equipment (column 5, line 48 - column 8, 
line 22 and figures 3 and 4); 

Falcon et al., US 6,632,094 Bl, (previously cited) teaches synchronized text 
highlighting with audio narration for improving reading skills (abstract and 
column 8, lines 23-26); 
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Sastry et a I., US 6,687,877 Bl , (previously cited) teaches content 
annotation synchronized with user actions and recorded audio (column 5, lines 
39-61); 

Blume, US 6,915,103 B2, (previously cited) teaches an audio broadcast 
system to read an audio track corresponding to the position and speed of a 
handheld stylus (column 3, line 32-56 and figures 1-5); 

Lilienthal, US 6,933,928 Bl, (previously cited) teaches a paperless book with 
visual text that advances as the audio text is finished (column 3, lines 43-50); 

Gaddy etal., US 7,228,189 Bl, (previously cited) teaches a method for 
synchronizing a streaming music file with a locally captured audio input 
(abstract, column 7, line 39 - column 8, line 36, and figures 3a and 3b); 

Estes et al., US 4,941,829 A, (previously cited) teaches a method for 
providing a dynamic tutorial for a computer program (see abstract); 

Ricard, US 5,745,738 A, (previously cited) teaches another method for 
providing simulations to demonstrate the use of software (see abstract); and 

Slye et al., US 5,395,242 A, teaches a method for recording a computer 
simulation, such as a video game simulation (see column 1, line 44 - column 2, 
line 7). 

Any inquiry concerning this communication or earlier communications 
from the examiner should be directed to DANIEL R. SELLERS whose telephone 
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number is (571)272-7528. The examiner can normally be reached on Monday to 
Friday, 9am to 5:30pm. 

If attempts to reach the examiner by telephone are unsuccessful, the 
examiner's supervisor, Vivian C. Chin can be reached on (571)272-7848. The fax 
phone number for the organization where this application or proceeding is 
assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from 
the Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR 
only. For more information about the PAIR system, see http://pair- 
direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If 
you would like assistance from a USPTO Customer Service Representative or 
access to the automated information system, call 800-786-9199 (IN USA OR 
CANADA) or 571 -272-1 000. 



/Daniel R. Sellers/ 
Examiner, Art Unit 2614 

/VIVIAN CHIN/ 

Supervisory Patent Examiner, Art Unit 2614 



